Sp6 to Sp7 add missing constraints and tables - #7643
Conversation
Triggered by 17da256 on branch refs/heads/issue-7551
|
I've added fixes for all of the unit tests that were failing in main. Currently, there are some unit tests failing specifically because of the new tables added from Specify 6 that have multi-field primary keys. This is causing issues with our django, datamodel, and sqlalchemy code. Working through different options for workarounds. |
|
Got the many-to-many fields working in the Django model, and got the unit tests working. Ready for review. |
|
Looks good, thanks for removing those unused Specify 6 tables 👍 |
See related (and first) #7683
|
Hey everyone! While I was testing this PR, I ran into an issue with databases created from the Specify 6. Specifically, the issue was caused due to Django expecting and using the Primary Key from the model when using the relationships, which does not exist in the many to many tables from Specify 6. To fix this, I moved the many to many code to its own migration outside of the initial migration and handled migrating the tables from Specify 6 to the new consistent format. See the changes in a788c8c..43bc97d. Some additional information about the migration is available in the migration itself: specify7/specifyweb/specify/migrations/0043_normalize_many_to_many.py Lines 25 to 37 in 355dda1 To facilitate the migration, we utilize Redis to store the old values of the Many to Many tables during the time the older tables are torn down and reconstructed. The stored records are then later populated into the new tables. specify7/specifyweb/specify/migrations/0043_normalize_many_to_many.py Lines 11 to 23 in 355dda1 Another caveat with this storage procedure is its memory usage. |
|
I tested out the many-to-many migration and reverting. I think the use of redis for retaining the migration is a good idea for this case, instead of something like a global variable in Django. The migration should only ever run once, so the risk of a failed migration and losing the information the docker redis instance being stopped or restarted is low. I played around with trying to replace some of the raw SQL in the many-to-many migration, but it was getting too hacky and complex. I think the current solution will be sufficient 👍 Let's go ahead and approve and merge this PR for the v7.12 release. |
Fixes #7551
Fixes #7617
Fixes #7626
This PR adds Django model definitions, constraints, and migrations for several legacy join tables and related entities that were present in existing Specify databases but missing from freshly initialized Sp7 databases.
These changes are based on a systematic comparison between an existing production database schema and a newly created schema, with the goal of ensuring that new databases accurately reflect the constraints and relationships relied on by legacy data and workflows.
Here is a link to an example of the difference between an Sp6 and Sp7 created database schema dump: https://www.diffchecker.com/qdMfXJCj/
After analyzing many different schema dumps, the schema differences between databases creation in Sp6 and Sp7, the following was found for constant differences:
16 Missing Foreign Key Constraints:
0 missing unique constraints were found, actually Sp7 create database had a few extra unique constrains compared to Sp6 created databases.
32 Missing Primary Key Constraints (Mostly due to unused tables not used in Sp7):
Missing / changed PRIMARY KEYs: 35
The following tables were identified in the schema diff but were not added in this PR because they are legacy, client specific, or no longer used by current Specify workflows:
These tables appear to not be required for new database creation or normal application operation, but let me know if any of these should be added.
Checklist
self-explanatory (or properly documented)
Testing instructions
mariadb-dump -uroot -proot --no-data db_name > dbname_schema.sql